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ABSTRACT 

In  this  paper  we  introduce  a  method  for  exploiting  entities 
from  the  emerging  Web  of  Data  for  enhancing  various  In¬ 
formation  Retrieval  (IR)  services.  The  approach  is  based 
on  named-entity  recognition  applied  in  a  set  of  search  re¬ 
sults,  and  on  a  graph  of  documents  and  identified  entities 
that  is  constructed  dynamically  and  analyzed  stochastically 
using  a  Random  Walk  method.  The  result  of  this  analysis 
is  exploited  in  two  different  contexts:  for  automatic  query 
expansion  and  for  re-ranking  a  set  of  retrieved  results.  Eval¬ 
uation  results  in  the  2015  TREC  Clinical  Decision  Support 
Track  illustrate  that  query  expansion  can  increase  recall  by 
retrieving  more  relevant  hits,  while  re-ranking  can  notably 
improve  the  ranked  list  of  results  by  moving  relevant  but 
low-ranked  hits  in  higher  positions. 

1.  INTRODUCTION 

The  Web  has  now  evolved  to  an  information  space  where 
both  unstructured  documents  (e.g.  Web  pages)  and  struc¬ 
tured  data  (e.g.  Linked  Open  Data  (LOD)  [8])  coexist  in  var¬ 
ious  forms.  An  important  observation  is  that  entity  names 
(like  names  of  persons,  locations,  etc.)  occur  in  all  kinds  of 
artifacts:  Web  pages,  database  cells,  RDF  triples,  etc.  A 
generic  hypothesis  that  we  investigate  is  whether  and  how 
we  can  exploit  named-entities  for  integrating  documents  (ac¬ 
tually  search  results)  with  data  and  knowledge.  The  idea  is 
to  construct  dynamically  a  graph  of  documents  and  entities, 
and  then  to  analyze  it  stochastically  using  a  Random  Walk- 
based  method.  Specifically,  we  model  the  search  process  as 
a  random  walker  of  the  graph  defined  by  the  top  documents 
returned  by  a  search  system  and  the  entities  identified  in 
these  documents.  For  analyzing  the  graph  and  scoring  its 
nodes,  we  exploit  both  the  ranking  of  the  returned  search 
hits,  the  “importance”  (within  the  search  context)  of  the  ex¬ 
tracted  entities  and  their  connectivity.  The  result  of  this 
analysis  is  exploited  in  two  different  contexts: 

•  for  automatic  query  expansion ,  aiming  to  construct 
and  submit  a  new  query  that  can  retrieve  more  rel¬ 
evant  hits 
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•  for  re-ranking  the  set  of  retrieved  results,  aiming  to 
promote  low-ranked  but  relevant  hits  referring  impor¬ 
tant  (for  the  search  context)  entities 
Figure  1  depicts  the  steps  of  the  analysis  process.  At  first, 
the  user  submits  a  query  to  a  search  system  and  the  top-L 
(e.g.  L  =  1,000)  results  are  retrieved.  Then,  Named  Entity 
Recognition  (NER)  is  applied  in  these  results  for  identifying 
LOD  entities.  In  the  next  (optional)  step,  more  semantic 
information  about  the  identified  entities  is  retrieved  from 
the  LOD  (like  properties  and  related  entities).  A  graph 
of  search  results  and  (semantically-enriched)  entities  is  con¬ 
structed  and  analyzed  stochastically.  The  document  and  en¬ 
tity  scores  (given  by  the  probabilistic  analysis)  are  exploited 
for  enhancing  various  IR  services,  e.g.  for  query  expansion 
and  results  re-ranking.  Finally,  the  user  interacts  with  the 
results. 


Figure  1:  The  steps  of  the  analysis  process. 

We  evaluated  the  effectiveness  of  the  proposed  method  in 
the  medicine  domain,  in  the  context  of  the  2015  TREC  Clin¬ 
ical  Decision  Support  Track.  The  results  showed  that:  (i) 
query  expansion  increases  the  number  of  relevant  hits  for  the 
majority  of  topics  (about  +70%  in  average)  (ii)  re-ranking 
improves  the  ranked  list  of  results  returned  by  a  classical 
IR  system  for  the  majority  of  topics  (about  +33%  average 
increment  in  infNDCG  and  +47%  in  P@10)  (iii)  additional 
semantic  information  about  the  entities  (properties  and  re¬ 
lated  entities)  can  affect  negatively  the  re-ranking  process. 

Related  Work.  The  work  in  [4,6]  introduced  an  exploratory 
search  process  based  on  extracted  entities  in  which  the  search 
results  are  connected  with  data  and  knowledge  at  query  time 
with  no  human  effort.  To  make  such  a  service  feasible  for 
large  amounts  of  data,  [9]  details  a  distributed  computation 


model  and  shows  how  the  required  computational  tasks  can 
be  factorized  and  expressed  as  MapReduce  functions.  For 
identifying  the  semantic  information  (entities  and  proper¬ 
ties)  that  better  characterizes  the  search  results,  the  work 
in  [5,  7]  introduces  a  Random  Walk-based  ranking  model 
that  exploits  both  the  ranking  of  the  returned  results,  the 
extracted  named  entities  and  their  connectivity,  and  which 
is  exploited  for  producing  and  showing  to  the  user  top-K 
semantic  graphs  related  to  the  search  results.  In  this  work, 
we  continue  this  line  of  research  and  we  investigate  whether 
and  how  such  “overview”  information  (entities  detected  in 
the  search  results  and  semantic  information  associated  to 
these  entities)  can  be  exploited  for  re-ranking  a  list  of  re¬ 
sults  as  well  as  for  query-expansion. 

2.  STOCHASTIC  ANALYSIS 

Query  and  retrieved  documents.  At  first,  the  user  sub¬ 
mits  a  query  q  describing  an  information  need  to  a  search 
system.  Let  L  be  the  number  of  top  documents  (hits)  to 
retrieve,  A  the  set  of  these  top-L  documents,  and  score(a) 
the  score  (value  in  the  range  [0, 1])  of  a  document  a  €  A. 
Moreover,  let  P  be  the  set  of  different  parts  that  constitute 
a  document  (e.g.  P  =  { title ,  abstract ,  body}),  ap  the  part  of 
document  a  £  A  of  type  p  £  P  (e.g.  its  abstract ),  and  w(p) 
the  weight  expressing  the  importance  of  a  part  p  £  P,  where 

E  P'€pw(p')  =  i- 

Extracted  entities.  Now,  a  LOD-based  NER  system  (e.g. 
X-Link  [3]  or  DBpedia  Spotlight  [11])  is  exploited  for  iden¬ 
tifying  entities  of  interest  (names  of  entities  that  are  im¬ 
portant  in  the  application  context)  in  all  parts  of  each  re¬ 
trieved  document.  A  list  of  identified  entities  is  derived. 
Each  entity  is  accompanied  by  its  URI  in  a  semantic  knowl¬ 
edge  base,  e.g.  DBpedia  [10].  In  more  details,  let  ent(ap) 
be  the  set  of  entities  identified  in  the  part  p  of  a  document 
a  £  A,  and  ent(a)  =  U p^pent(ap)  the  set  of  all  entities  iden¬ 
tified  in  a  document  a  (in  all  its  parts).  E  =  U aeAent(a) 
is  the  set  of  all  entities  identified  in  A.  Conversely,  let 
docs(e)  =  {  a  £  A  \  e  £  ent(a)}  be  the  elements  of  A  in 
which  e  has  been  identified.  Finally,  let  e/(e,  ap)  be  the  fre¬ 
quency  (number  of  occurrences)  of  the  entity  e  in  the  part 
p  of  the  document  a. 

The  importance  of  an  entity  e  identified  in  a  document 
a  is  defined  as:  imp(e,a)  =  E  '  w(p))- 

peP  e/  G ent(op ) 

The  score  takes  into  account  both  the  number  of  entity  oc¬ 
currences  and  the  part(s)  in  which  the  entity  has  been  iden¬ 
tified.  In  the  entire  set  of  top-L  documents,  the  importance 
of  an  entity  e  is  defined  as: 

ImpScore(e)  =  (imp(e,  a)  •  score(a))  (1) 

a£docs(e) 

We  notice  that  the  score  is  higher  if  e  has  been  identified 
in  the  top  scored  documents.  This  entity  importance  score  is 
actually  a  variation  of  the  formula  proposed  in  [4]  for  scoring 
entities  identified  in  a  set  of  search  results. 

Graph  construction.  We  first  construct  a  semantic  graph 
of  documents  and  entities,  denoted  by  A.  We  consider 
both  the  documents  and  the  entities  as  vertices  in  A,  while 
for  drawing  the  edges  we  take  into  account  in  which  doc¬ 
uments  an  entity  was  identified.  Specifically,  we  draw  an 


edge  starting  from  an  entity  e  and  ending  to  a  document  a, 
if  e  £  ent(a ),  i.e.  e  has  been  identified  in  a.  Now,  by  exploit¬ 
ing  a  semantic  knowledge  base,  we  fetch  interesting  (for  the 
search  context)  triples  that  describe  information  about  the 
identified  entities,  like  properties  and  related  entities  (recall 
that  each  identified  entity  is  accompanied  by  its  URI  in  a 
semantic  knowledge  base).  We  enrich  the  graph  with  the 
corresponding  properties,  entities  and  associations.  Let  R 
be  this  set  of  related  properties  and  entities  (not  identified 
in  the  search  results). 

Now  we  transform  the  graph  to  a  State  Transition  Graph 
(STG),  denoted  by  Q  =  (£,V).  We  do  that  by  simply  con¬ 
sidering  also  the  opposite  direction  for  each  directed  edge. 
In  our  context,  we  consider  that  if  a  property  connects  two 
nodes  in  A,  then  these  nodes  are  semantically  biconnected. 
For  example,  in  the  case  of  a  document  a  and  an  entity 
e  we  can  either  say  that  (e,  “identifiedln”,  a)  or  that  (a, 
“ contains ”,  e),  i.e.  the  difference  lies  in  how  we  name  the 
property. 

Edge  Weighting.  For  weighting  the  edges,  we  consider  the 
document  and  entity  scores.  Specifically,  the  edge  weight 
from  a  node  n  to  a  connected  node  n  is  defined  as: 


ImpScore(n) 

E}  ImpScore(e') 
e'  Eent(n'  ) 


n'  £  A,  n  £  E 
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where  p  is  the  probability  the  random  walker  to  select  a 
document-node  when  being  at  an  entity-node  (note  that  the 
weights  of  the  outgoing  edges  of  a  single  node  must  represent 
transition  probabilities,  i.e.  they  must  sum  to  1).  We  notice 
that  when  the  walker  lies  in  a  document-node,  the  transition 
probabilities  are  affected  by  the  importance  of  the  connected 
entities.  Specifically,  the  higher  the  score  of  an  entity  is,  the 
higher  is  the  probability  to  move  to  that  entity.  Similarly,  in 
case  the  walker  lies  in  an  entity-node,  the  transition  prob¬ 
abilities  to  document-nodes  are  affected  by  the  document 
scores,  while  the  transition  probabilities  to  properties  and 
related  entities  are  defined  equiprobably.  Finally,  when  the 
walker  lies  in  a  related  property /entity  node,  he  can  move 
to  the  connected  entity-nodes  equiprobably. 


Analyzing  the  STG.  The  objective  is  to  find  the  proba¬ 
bility  the  random  walker  to  be  in  a  specific  node.  We  ana¬ 
lyze  the  STG  using  a  PageRank-like  scoring  formula.  For  a 
node  n,  let  in(n )  be  the  set  of  nodes  that  point  to  n.  The 
PageRank-like  value  r(n)  is  defined  as: 


r(n)  =  d-  Jump{n)-\-{l—  d)  E  (weightin'  — >  n )  •  r(n'))  (3) 

n'  £in(n) 

where  d  is  the  probability  (decay  factor)  the  walker  to  per¬ 
form  a  random  jump,  Jump(n)  expresses  the  probability 
the  walker  to  jump  to  the  node  n,  and  weight(n  — »•  n) 
is  the  probability  (as  defined  in  Formula  2)  the  walker  to 
visit  n  when  being  in  a  node  n  connected  to  n.  As  re¬ 
gards  Jump(n ),  we  allow  the  random  jumps  only  to  nodes 


corresponding  to  documents,  i.e.  Jump(n)  =  0  if  n  £  A. 
In  addition,  we  adjust  the  jump  probabilities  according  to 
the  document  scores  (instead  of  assuming  a  uniform  distri¬ 
bution).  Specifically,  for  a  node  n  £  A  we  consider  the 
following  formula  for  the  random  jumps: 

T  ,  x  scorein )  , 

Jump(n)  =  ~  (i\  (4) 

2^  score(a') 

a'  £  A 

PageRank  requires  some  initial  values  for  the  graph  nodes. 
We  can  define  a  uniform  distribution.  Specifically,  for  each 
node  n  we  set  r(n )  =  1/|£|  (S  is  the  set  of  STG  nodes). 
Finally,  the  values  are  computed  iteratively  and  iterations 
should  be  run  to  convergence.  According  to  [12],  the  number 
of  iterations  required  for  convergence  is  empirically  O(logra), 
where  n  is  the  number  of  edges. 

Exploiting  the  Outcome.  After  running  the  above  algo¬ 
rithm,  all  nodes  receive  a  PageRank-like  score.  The  higher 
the  score  of  a  node  is,  the  most  important  (and  relevant  to 
the  search  context)  that  node  is  considered. 

Query  Expansion.  We  exploit  the  top-scored  entities  (e.g. 
the  top- 10)  for  expanding  the  query  string.  Note  that  an 
identified  entity  may  actually  be  a  synonym  (or  scientific 
name)  of  a  term  in  the  query,  or  in  the  case  of  medical 
records,  a  case  narrative  may  actually  correspond  to  a  par¬ 
ticular  disease,  thus  including  this  entity  name  in  the  query 
string  may  invoke  the  retrieval  of  new  relevant  documents 
(that  did  not  exist  in  the  initial  list  of  results),  while  some 
relevant  documents  may  be  moved  in  higher  ranks. 
Re-ranking  the  Retrieved  Results.  All  document-nodes  have 
received  a  final  PageRank  score.  We  exploit  these  scores  for 
re-ranking  the  list  of  retrieved  results.  Low-ranked  docu¬ 
ments  referring  highly-scored  entities  will  now  receive  a  high 
score  and  will  be  promoted  in  the  new  ranked  list  of  results. 

3.  EVALUATION 

3.1  Corpus  and  Setup 

We  evaluated  the  proposed  approach  in  the  2015  TREC 
Clinical  Decision  Support  track1.  The  track  focuses  on  re¬ 
trieving  biomedical  articles  relevant  for  answering  generic 
clinical  questions  about  medical  records.  We  used  Apache 
Lucene2  for  indexing  the  collection  while  we  indexed  the  ti¬ 
tle ,  the  abstract  and  the  body  of  each  document. 

As  regards  the  topics,  each  one  is  a  medical  case  narrative 
serving  as  an  idealized  representation  of  an  actual  medical 
record  and  it  describes  information  such  as  the  patient’s 
medical  history,  current  symptoms,  etc.  For  each  provided 
topic,  an  effective  IR  system  must  find  documents  that  can 
help  the  physician  to  answer  a  common  generic  clinical  ques¬ 
tion  such  as  what  is  the  patient’s  diagnosis  or  what  tests 
should  the  patient  receive  based  on  the  medical  report.  The 
provided  30  topics  are  annotated  according  to  the  three  most 
common  generic  clinical  question  types  [1]:  diagnosis  (what 
is  the  patient’s  diagnosis  based  on  the  medical  report),  test 
(what  tests  should  the  patient  receive  based  on  the  medical 
report),  treatment  (how  should  the  patient  be  treated  based 
on  the  medical  report).  The  first  10  topics  are  of  type  diag¬ 
nosis. ,  the  next  10  topics  are  of  type  test ,  while  the  last  10 
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are  of  type  treatment.  Finally,  for  each  topic  a  description 
and  a  (smaller)  summary  is  given. 

For  performing  NER  in  the  indexed  fields  of  top  retrieved 
documents  we  used  X-Link.  X-Link  [2,  3]  is  a  configurable, 
LOD-based  NER  system  capable  to  identity  entities  in  a  doc¬ 
ument,  link  the  identified  entities  with  semantic  resources, 
and  enrich  them  with  additional  semantic  information  com¬ 
ing  from  external  semantic  knowledge  bases.  As  the  enti¬ 
ties  of  interest,  we  used  diseases,  drugs,  proteins,  and  chem¬ 
ical  substances  coming  from  DBpedia,  while  for  testing  the 
case  of  entity  enrichment,  we  used  the  DBpedia  dct :  sub  j  ect 
property3 . 

3.2  Submitted  Runs 

We  submitted  the  following  4  runs  for  testing  the  proposed 
re-ranking  approach  using  different  parameters: 

•  (RRdO)  Result  re-ranking  with  topic  description  as  the 
query,  L  =  250,  d  =  0.0,  p  =  1.0 

•  (RRdO+EE)  Result  re-ranking  with  topic  description  as 
the  query,  L  =  250,  d  =  0.0,p  =  0.7  (entity  enrichment 
is  applied) 

•  (RRd2)  Result  re-ranking  with  topic  description  as  the 
query,  L  =  250,  d  =  0.2,  p  =  1.0 

•  (RRsO)  Result  re-ranking  with  topic  summary  as  the 
query,  L  =  250,  d  =  0.0,  p  =  1.0 

We  submitted  1  run  for  testing  query  expansion  on  the  re¬ 
ranked  list  returned  by  Lucene: 

•  (RRdO+QE)  Query  expansion  with  the  top-10  scored 
entities,  using  topic  description  as  the  initial  query, 
L  =  250,  d  =  0.0,  p=l.O 

We  also  submitted  1  run  for  testing  the  effect  of  re-ranking 
on  the  list  returned  by  Lucene  after  query  expansion. 

•  (RRdO+QE+RRdO)  Query  expansion  using  the  top- 10  en¬ 
tities,  then  stochastic  re-ranking,  using  the  topic  de¬ 
scription  as  the  initial  query,  L  =  250,  d  =  0.0,p=1.0 

3.3  Results 

At  first  we  should  point  out  that  the  top- 1000  initial  list 
returned  by  Lucene  contains  in  average  40  relevant-for-sure 
hits  (almost  the  same,  in  average,  using  either  the  topic 
description  or  the  topic  summary  as  the  submitted  query). 
This  means  that  Lucene  did  not  manage  to  retrieve  many 
relevant-for-sure  documents  in  the  top- 1000  list.  This  en¬ 
forces  the  need  for  an  effective  query  expansion  approach 
that  can  retrieve  more  relevant  hits,  or  for  an  effective  re¬ 
ranking  approach  that  can  bring  these  few  relevant-for-sure 
documents  in  higher  positions  in  the  returned  ranked  list. 

3.3.1  Query  Expansion  Effect 

Query  expansion  managed  to  retrieve  more  relevant  hits 
for  the  majority  of  topics.  Specifically,  the  number  of  rel¬ 
evant  hits  was  increased  for  18/30  topics,  was  reduced  for 
9/30  topics,  and  remained  the  same  for  3/30  topics.  In  av¬ 
erage,  the  number  of  relevant  hits  was  increased  about  70% 
in  these  18  topics.  Figure  2  depicts  the  results  for  all  30 
topics.  We  notice  that  for  some  topics  the  improvement  is 
very  large,  e.g.  for  a  test  topic,  query  expansion  managed 
to  retrieve  +87  relevant  hits  and  for  a  diagnosis  topic  +61 

3In  DBpedia,  the  dct :  subj  ect  property  (http :  //purl .  org/ 
dc/terms/subject)  provides  categories/groups  in  which  the 
corresponding  entity  belongs. 


relevant  hits.  Such  improvement  in  recall  may  be  very  im¬ 
portant  for  search  applications  in  professional  domains  (like 
in  the  medicine  domain)  where  the  main  goal  is  to  retrieve 
almost  all  documents  that  are  relevant  to  an  issue. 


Figure  2:  Query  expansion  effect  on  number  of  relevant  re¬ 
trieved  results  per  topic 


3.3.2  Re -Ranking  Effect 

As  regards  the  effect  of  re-ranking  in  the  list  of  results  de¬ 
rived  by  the  query  expansion  approach,  re-ranking  improved 
the  list  of  results  for  the  majority  of  topics.  Specifically,  as 
regards  the  infNDCG  evaluation  metric,  the  value  was  in¬ 
creased  for  20/30  topics,  was  reduced  for  7/30  topics  and 
remained  the  same  for  3/30  topics.  As  regards  the  P@10  met¬ 
ric,  the  value  was  increased  for  11/30  topics,  was  reduced  for 
2/30  topics  and  remained  the  same  for  17/30  topics.  For  the 
cases  in  which  we  had  improvement,  infNDCG  was  increased 
about  33%  in  average,  while  P@10  about  47%.  These  results 
show  that  the  proposed  re-ranking  method  managed  to  move 
relevant  but  initially  low-ranked  hits  in  higher  positions  for 
the  majority  of  topics.  Figures  3  and  4  depict  the  results  for 
all  topics. 

3.3.3  Entity  Enrichment  Effect 

Entity  enrichment  affected  negatively  the  results  for  the 
majority  of  topics.  Specifically,  as  regards  the  infNDCG  eval¬ 
uation  metric  the  value  was  increased  for  10/30  topics,  re¬ 
duced  for  17/30  topics,  and  remained  the  same  for  3/30 
topics.  As  regards  the  P@10  metric,  the  value  was  increased 
for  4/30  topics,  reduced  for  12/30  topics,  and  remained  the 
same  for  14/30  topics.  This  means  that  the  specific  semantic 
information  about  the  identified  entities  (DBpedia  subject 
property)  can  mislead  the  random  walker  and  affect  nega¬ 
tively  the  re-ranking  of  the  retrieved  results.  Figures  5  and 
6  depict  the  full  results. 

3.3.4  Description  vs  Summary 

We  compared  the  number  of  relevant  retrieved  hits  when 
using  the  topic  description  as  the  submitted  query  and  when 
using  the  topic  summary.  Figure  7  depicts  the  results.  When 
using  topic  summary,  the  number  of  relevant  hits  is  increased 
for  13  topics  while  it  is  reduced  for  14  topics.  We  also  notice 
that  for  some  topics  (for  the  topics  1,  4,  8,  16,  21,  22)  the 
number  of  relevant  hits  in  each  approach  differs  significantly. 
Specifically,  for  the  topics  1  and  16,  description  retrieved 


■  RRdO+QE  □  RRdO+QE+RRdO 


1  3  5  7  9  11  13  15  17  19  21  23  25  27  29 

Topic 

Figure  3:  Re-ranking  effect  on  infNDCG  per  topic 


Figure  4:  Re-ranking  effect  on  P@10  per  topic 
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Figure  5:  Entity  enrichment  effect  on  infNDCG  per  topic 


1  3  5  7  9  11  13  15  17  19  21  23  25  27  29 

Topic 

Figure  6:  Entity  enrichment  effect  on  P@10  per  topic 


more  relevant  hits,  while  for  the  topics  4,  8,  21,  22  summary 
performed  better.  From  the  above  results  we  cannot  infer  a 
safe  conclusion  about  which  topic  part  is  better  to  use  for 
querying  the  underlying  search  system. 


Topic 

Figure  7:  Submitted  query:  description  vs  summary 


3.3.5  Comparison  with  the  other  TREC  systems 

As  regards  the  comparison  with  the  other  systems  that 
participated  in  the  same  TREC  track,  at  first  we  should 
stress  that,  in  our  context,  such  a  comparison  is  not  impor¬ 
tant  because  we  focus  on  how  to  improve  an  existing  list  of 
results  returned  by  another  search  system,  i.e.  we  act  in  a 
meta-search  level.  Nevertheless,  here  we  present  the  results 
for  the  RRdO  run  (the  results  are  similar  for  all  runs).  As  re¬ 
gards  the  evaluation  metric  infNDCG,  our  system  was  above 
average  for  11/30  topics,  below  average  for  18/30  topics,  and 
same  as  average  for  1  topic.  As  regards  the  metric  P@10,  our 
system  was  above  average  for  10/30  topics,  below  average  for 
14/30  topics,  and  same  as  average  for  6/30  topics.  Figures 
8  and  9  depict  the  results  for  all  topics. 


4.  CONCLUSION 

We  have  proposed  a  generic  entity-based  approach  for 
query  expansion  and  results  re-ranking.  The  approach  is 
based  on  named  entity  recognition  applied  in  a  set  of  re¬ 
trieved  documents,  and  on  a  graph  of  documents  and  entities 
that  is  constructed  dynamically  and  analyzed  stochastically. 
Experimental  results  in  the  2015  TREC  Clinical  Decision 
Support  Track  showed  that:  i)  query  expansion  can  notably 
increase  recall  by  retrieving  more  relevant  hits  (70%  more 
relevant  hits  in  our  experiments),  ii)  re-ranking  can  improve 
the  ranked  list  of  results  returned  by  a  classical  IR  system 
by  moving  low-ranked  but  relevant  hits  in  higher  positions 
(30%  increase  in  infNDCG  and  47%  in  P@10  in  our  experi¬ 
ments),  and  iii)  additional  semantic  information  about  the 
entities  (like  properties  and  related  entities)  can  affect  neg¬ 
atively  the  re-ranking  process  and  thus  must  be  carefully 
considered  during  the  stochastic  analysis. 

In  future  we  plan  to  perform  a  more  extensive  evaluation 
using  more  topics.  Our  aim  is  to  infer  under  what  circum¬ 
stances  the  proposed  approach  is  effective  and  enhances  IR, 
or  ineffective  and  thus  must  be  avoided. 


Figure  8:  Difference  from  median  infNDCG  per  topic 


Figure  9:  Difference  from  median  P@10  per  topic 
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